A Sound MagicBoard

Vision and audio can be used in a complementary way to efficiently create augmented reality tools. This paper describes how sound detection can be used to enhance the efficiency of a computer vision augmented reality tool such as the MagicBoard. A MagicBoard is an ordinary white board with which a user can handle both real and virtual information, allowing, e.g., for a copy&paste operation on a physicaldrawing on the board. The electronic part of the MagicBoard consists of a video–projector and the concurrent use of several detectors such as a camera and microphones. In our system, sound is used to support a purely vision– based finger tracker, especially during the most critical phase, i.e., when trying to detect and localize a finger tap as a click on the board. The relative phase of a signal caused by a finger tap on the board, detected by several microphones is used to estimate the postition of the finger tap. In the probable case of several possible estimates, the estimates are eventually verified by the visual tracker. The resulting system is surprisingly precise and robust.